Repository navigation
Conversation
haffi96
marked this pull request as draft
October 27, 2026 09:20
Author
|
The main motivation here is to enable the H265 pre-encoded ingest path with length prefixed NAL units more or less trying to mirror Livekit Go SDK pre-encoded publish path Along the way, benchmarking the changes against a Go publisher client, I found that there was differences in the pacer logic in libwebrtc and some smaller gains in the publish path that could be made. I've addressed those here in a separate commits so they can easily be excluded if risky This PR can probably be broken down but I chose to put them all in one as they go hand in hand for testing/verification. Happy to break this down into a smaller piece or whatever is preferable |
Pre-encoded H.265 could only be ingested as Annex-B, and a single-layer pre-encoded H.265 track could not be published at all. - livekit-capture: `LengthPrefixedAccessUnitParser` (was the test-only AVC parser) handles H.264 (avc3) and H.265 (hev1) length-prefixed streams, with an opt-in single-slice flush that completes an access unit as soon as its slice arrives instead of a frame later. - livekit-capture: `access_unit_from_h265_hvcc` mirrors `access_unit_from_h264_avc`; the GStreamer source accepts `hev1` caps. - livekit-capture: `frame_metadata_from_sei` reads the LKTS packet-trailer SEI (the Go SDK `ReaderTrackWithPacketTrailer` convention) for the pump's frame metadata callback. - livekit: announce the codec of single-layer video tracks in AddTrackRequest. Without it the SFU answered with its default (VP8), so a pre-encoded H.265 track negotiated VP8 and the pass-through dropped every access unit. The Go SDK always sends it. - local_video: `encoded_publisher` publishes a length-prefixed stream received over TCP. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
libwebrtc paces packets at 2.5x its bandwidth estimate, which starts at
1 Mbps for camera tracks. A pre-encoded stream arrives at the bitrate its
upstream encoder chose and cannot slow down, so for the first ~9 s frames
queued in the pacer: up to ~420 ms (Rust) / ~570 ms (C++) at p99 for a
1080p30 ~5 Mbps H.265 stream, and 16-50 ms after each keyframe in some
runs once the estimate settled.
- webrtc-sys: read field trials from LIVEKIT_WEBRTC_FIELD_TRIALS
("Name/Value/" format), taking precedence over the SDK's own. With
"WebRTC-Video-Pacing/factor:10/" startup p99 drops to ~1.5-2.2 ms.
- livekit: exempt pre-encoded tracks from the 1 Mbps cap on
x-google-start-bitrate, like screen share (halves the startup penalty
on its own).
- local_video: `encoded_publisher --stats` logs the pacer delay per
packet (totalPacketSendDelay / packetsSent), target bitrate and
bandwidth estimate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- livekit-capture: the length-prefixed parser copied each access unit twice and rescanned it byte by byte for start codes to classify the frame (~205 us per 1080p access unit). It now converts in place during one copy and classifies from the NAL ranges it already has (~2.5 us). find_start_code jumps between 0x01 bytes instead of matching at every offset, which speeds up every Annex-B path; an exhaustive test pins it to the old behaviour. Steady-state publish latency p50 1.21 -> 0.81 ms. - livekit: publish pre-encoded video without simulcast. FFI clients such as the C++ SDK leave simulcast on by default; the track then announced 360p/180p layers whose senders forwarded copies of the full stream, requested keyframes on almost every frame, and competed with the real layer in the pacer. - local_video: `encoded_publisher` counts keyframe requests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
haffi96
force-pushed
the
feat/h265-pre-encoded-length-prefixed
branch
from
October 30, 2026 09:27
9a574e8 to
b94d889
Compare
haffi96
marked this pull request as ready for review
October 30, 2026 13:57
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hardware encoder bridges can often emit H.265 as 4-byte length-prefixed NAL units (
hev1), which the Rust SDK could not publish pre-encoded. Once publishing worked, these tracks also sat in the pacer for seconds after publishing, and the publish path did redundant work on every frame. This PR fixes all three, one commit per stage, each with its own changeset:1. Publish length-prefixed pre-encoded H.265
livekit-capture:LengthPrefixedAccessUnitParser(previously test-only, H.264 only) handles H.264 and H.265, with an opt-in single-slice flush that completes a picture as soon as its slice arrives instead of a frame later. Addsaccess_unit_from_h265_hvcc,hev1caps in the GStreamer source, andframe_metadata_from_seifor the LKTS packet-trailer SEI (the Go SDK'sReaderTrackWithPacketTrailerconvention).livekit: single-layer video tracks announce their codec insimulcast_codecs, as the Go SDK does.local_video: newencoded_publisherexample that publishes a length-prefixed stream received over TCP.2. Let pre-encoded tracks leave the pacer promptly
A pre-encoded stream arrives at its encoder's bitrate, but the starting estimate is capped at 1 Mbps, so packets queued in the pacer for ~8 s after publishing (~510 ms per packet at the peak for 1080p30 at 4 Mbps).
webrtc-sys: read field trials fromLIVEKIT_WEBRTC_FIELD_TRIALS. WithWebRTC-Video-Pacing/factor:10/the startup delay drops to ~4 ms, then ~0.livekit: pre-encoded tracks are exempt from the starting-estimate cap, like screen share (peak ~510 ms → ~210–440 ms on its own).encoded_publisher --statslogs codec, frames sent and pacer delay from standard WebRTC stats.3. Trim the publish path
find_start_codealso skips between0x01bytes, which speeds up every Annex-B path (an exhaustive test pins it to the old behaviour).Testing
cargo test -p livekit-capture --features source-gstreamer(55 pass),cargo test -p livekit --lib,cargo test -p webrtc-sys --lib.gst-launch-1.0 videotestsrc ! nvcudah265enc ! h265parse config-interval=-1 ! video/x-h265,stream-format=hev1,alignment=au ! tcpserversinkintoencoded_publisher --single-slice-flush --statsagainstlivekit-server --dev. Cross-checked end to end with a packet capture and a subscriber: steady-state publish latency p50 0.84 ms (Go SDK: 0.49 ms).--single-slice-flush --packet-trailertolivekit-server --devand LiveKit Cloud; the subscriber received 720p30 with the user timestamp and frame ID intact.Notes
Created with Claude Opus 5.5 in Claude Code.